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^ ' Abstract 

■ We propose an approximation of maximum-likelihood detection in ISI channels based on linear 

programming or message passing. We convert the detection problem into a binary decoding problem, 
which can be easily combined with LDPC decoding. We show that, for a certain class of channels and 
O , in the absence of coding, the proposed technique provides the exact ML solution without an exponential 

complexity in the size of channel memory, while for some other channels, this method has a non- 
diminishing probability of failure as SNR increases. Some analysis is provided for the error events of 
' the proposed technique under linear programming. 

(N ' 



■ I. Introduction 

O. 

Intersymbol interference (ISI) is a characteristic of many data communications and storage channels. 



o 



Systems operating on these channels employ error-correcting codes in conjunction with some ISI reduction 
•1— I ■ 

^ ■ technique, which, in magnetic recording systems, is often a conventional Viterbi detector It is known 

5^ , that some gain will be obtained if the equalization and decoding blocks are combined at the receiver by 

exchanging soft information between them. A possible approach to achieving this gain is to use soft- 
output equalization methods such as the BCJR algorithm [1] or the soft-output Viterbi algorithm (SOVA) 
[2] along with iterative decoders. However, both BCJR and SOVA suffer from exponential complexity in 
the length of the channel memory. 

Kurkoski et al. [3] proposed two graph representations of the ISI channel that can be combined with 
the Tanner graph of the LDPC code for message-passing decoding. Their bit-based representation of 
the channel contains many 4-cycles, which results in a significant performance degradation compared 
to maximum-likelihood (ML) detection. On the other hand, message passing (MP) on their state-based 
representation, where messages contain state rather than bit information, has a performance and overall 
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complexity similar to BCJR, while benefiting from a parallel structure and reduced delay. Among other 
works, Singla et al. [4] applied message passing on a bit-based graph representation of a two-dimensional 
ISI channel combined with an LDPC Tanner graph. However, similar to the case of one-dimensional ISI, 
the abundance of short cycles prevents the algorithm from performing close to optimal. 

Linear programming (LP) has been recently applied by Feldman et al. [5] to the problem of ML 
decoding of LDPC codes, as an alternative to MP techniques. In this method, the binary parity-check 
constraints of the code are relaxed to a set of linear constraints in the real domain, thus turning the 
integer problem into an LP problem. While LP decoding performs closely to MP algorithms such as the 
sum-product algorithm (SPA) and the min-sum algorithm (MSA), it is much easier to analyze for finite 
code lengths. 

Motivated by the success of LP decoding, in this work we study the problem of ML detection in the 
presence of ISI, which can be written as an integer quadratic program (IQP). We convert this problem into 
a binary decoding problem, which can be used for MP decoding, or, after relaxing the binary constraints, 
LP decoding. Furthermore, decoding an underlying LDPC code can be incorporated into this problem 
simply by including the parity checks of the code. 

By a geometric analysis we show that, in the absence of coding, if the impulse response of the ISI 
channel satisfies certain conditions, the proposed LP relaxation is guaranteed to produce the ML solution 
at all SNR values. This means that there are ISI channels, which we call LP-proper channels, for which 
uncoded ML detection can be achieved with a complexity polynomial in the channel memory size. On 
the other end of the spectrum, some channels are LP-improper, i.e. the LP method results in a nonintegral 
solution with a probability bounded away from zero at all SNR, even in the absence of noise. Furthermore, 
we observe some intermediate asymptotically LP-proper channels where the performance asymptotically 
converges to that of ML detection at high SNR. When message passing is used instead of LP, we observe 
a similar behavior. Moreover, when LDPC decoding is incorporated in the detector, LP-proper channels 
achieve very good performance, while some other channels cannot go below a certain word error rate 
(WER). 

The rest of this paper is organized as follows. In Section II, we describe the channel, and introduce the 
LP relaxation of ML detection. The performance analysis and simulation results of uncoded graph-based 
detection are presented in Section III. In Section IV, we study the combination of graph-based detection 
and LDPC decoding, and Section V concludes the paper. 
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Fig. 1. Binary-input ISI channel. 



II. Relaxation of the Equalization Problem 

A. Channel Model 

We consider a partial-response (PR) channel with bipolar (BPSK) inputs, as described in Fig. [U and 
use the following notation for the transmitted symbols. 

Notation 1: The bipolar version of a binary symbol, b G {0, 1}, is denoted by b G { — 1,1}, and is 
given by 

6=1-26. (1) 
The partial-response channel transfer polynomial is h{D) = X]f=o where fi is the channel 

memory size. Thus, the output sequence of the PR channel in Fig. [T] before adding the white Gaussian 
noise can be written as 

(2) 



yt = ^ hiXt-i. 

i=0 



B. Maximum-likelihood (ML) Detection 

Having the vector of received samples r = [ri r2 • • • r„]^, the ML detector solves the optimization 
problem 



Minimize — y||2 

Subject to X G 



(3) 



where C {0,1}"^ is the codebook and || • ||2 denotes the L2-norm. By expanding the square of the 
objective function, the problem becomes equivalent to minimizing 



^ r"^ - 2rt ^ Kxt-i + y^/i: 



Xt—iXt—j 



2 ~2 
i -^t—i 



(4) 
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where, for simplicity, we have dropped the limits of the summations. Equivalently, we can write the 
problem in a general matrix form 



1 



Minimize 

Subject to xe^, (5) 
where in this problem qt = Yli ^i'^^t+i, and P = H^H, with H defined as the n x n Toeplitz matrix 

ho 



H 



hp, ■ ■ 

h, 







ho 



ho 



h, 



he 



(6) 



Here we have assumed that n zeros are padded at the beginning and the end of the transmitted sequence, 
so that the trellis diagram corresponding to the ISI channel starts and ends at the zero state. If the signals 
are uncoded, i.e. 'io = {0, 1}"^, and q and P are chosen arbitrarily, © will represent the general form 
of an integer quadratic programming (IQP) problem, which is, in general, NP-hard. In the specific case 
of a PR channel, where we have the Toeplitz structure of the problem can be solved by the Viterbi 
algorithm with a complexity linear in n, but exponential in /i. However, this model can also be used to 
describe other problems such as detection in MIMO or two-dimensional ISI channels. Also, when the 
source symbols have a non-binary alphabet with a regular lattice structure such as the QAM and PAM 
alphabets, the problem can be reduced to the binary problem of © by introducing some new variables. 



C. Problem Relaxation 

A common approach for solving the IQP problem is to first convert it to an integer LP problem by 
introducing a new variable for each quadratic term, and then relax the integrality condition; e.g. see 
[6]. While this relaxed problem does not necessarily have an integer solution, it can be used along with 
branch-and-cut techniques to solve integer problems of reasonable size. A more recent method is based 
on dualizing the IQP problem twice to obtain a convex relaxation in the form of a semi-definite program 
(SDP) [7][8]. 

In this work, we use the linear relaxation due to the lower complexity of solving LPs compared to 
SDPs. Unlike in [6], where the auxiliary variables are each defined as the product of two 0-1 variables, we 
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define them as the product of ±1 variables, which, as we will see, translates into the modulo-2 addition 
of two bits when we move to the 0-1 domain. This relaxation is more suitable for our purpose, since 
modulo-2 additive constraints are similar to parity-check constraints; thus, message-passing decoders 
designed for linear codes can be applied without any modification. However, it can be shown that this 
relaxation gives the exact same solution as in [6]. 
To linearize (|4]), we define 

ztj = xt ■ xt-j, j = 1, . . . , /i, t = j + 1, . . . , n. (7) 

In the binary domain, this will be equivalent to 

zt,j ^ Xt® xt-j, (8) 

where © stands for modulo-2 addition. Hence, the right-hand side of Q is a linear combination of {xt} 
and {zt,j}, plus a constant, given that Xi^ = 1 is a constant. With some simplifications, the IQP in ([5]) 
can be rewritten as 

Minimize ^ qtXt + ^^^t 
t t J 

Subject to X G 

— ^t® Xt—ji j = Ij . . . , fl, 

t = j + l,...,n, (9) 

where, in the equalization problem, 

min(/i— j,n— 

Afj = -Pt,t-j = - ^ Kh^+j. (10) 

i=0 

In this optimization problem, we call {xi} the information bits, and {ztj} the state bits. It can be seen 
from (fTOl) that is independent of t, except for indices near the two ends of the block; i.e. 1 <t < /i 
and n — fi + l<t<n. In practice, this "edge effect" can be neglected due to the zero padding at the 
transmitter. For clarity, we sometimes drop the first subscript in Xtj, when the analysis is specific to the 
PR detection problem. 

The combined equalization and decoding problem © has the form of a single decoding problem, which 
can be represented by a low-density Tanner graph. Fig. [2] shows an example of the combination of a PR 
channel of memory size 2 with an LDPC code. We call the upper and lower layers of this Tanner graph 
the code layer and the PR layer (or the PR graph), respectively. The PR layer of the graph consists of 
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Fig. 2. PR channel and LDPC code represented by a Tanner graph. 



(in check nodes ctj of degree 3, each connected to two information bit nodes xt, xt-j, and one distinct 
state bit node, ztj. Also, the PR layer can contain cycles of length 6 and higher. If a coefficient, Xtj, is 
zero, its corresponding state bit node, ztj, and the check node it is connected to can be eliminated from 
the graph, as they have no effect on the decoding process. 

It follows from (fTOl) that the coefficients of the state bits in the objective function, {Xtj}, are only a 
function of the PR channel impulse response, while the coefficients of the information bits are the results 
of matched filtering the noisy received signal by the channel impulse response, and therefore dependent on 
the noise realization. Once the variable coefficients in the objective function are determined, LP decoding 
can be applied to solve a linear relaxation of decoding on this Tanner graph. We call this method LP 
detection. In the relaxation of [5], the binary parity-check constraint corresponding to each check node 
c is relaxed as follows. Let Nc be the index set of neighbors of check node c, i.e. the variable nodes it 
is directly connected to in the Tanner graph. Then, we include the following constraints 

- ^ x^<\V\-l, V y C TVc s.t. |y| is odd. (11) 

In addition, the integrality constraints Xi G {0, 1} are relaxed to box constraints < < 1. This 
relaxation has the "ML certificate property," i.e. if the solution of the relaxed LP is integral, it will also 
be the solution of ©. 

The coefficients in the linear objective function, after some normalization, can also be treated as log- 
likelihood ratios (LLR) of the corresponding bits, which can be used for iterative MP decoding. In this 
work, we have mostly used the Min-Sum Algorithm (MSA), since, similar to LP decoding, it is not 
affected by the uniform normalization of the variable coefficients in (|9]). 
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III. Performance Analysis of Uncoded Detection 

In this section, we study the performance of LP detection in the absence of coding, i.e. solving © with 
^ = {0, l}"^. It is known that if the off-diagonal elements of P are all nonpositive; i.e. ^ > 0, Vj ^ 0, t, 
the 0-1 problem is solvable in polynomial time by reducing it to the MIN-CUT problem; e.g. see [9]. 
As an example, Sankaran and Ephremides [10] argued using this fact that when the spreading sequences 
in a synchronous CDMA system have nonpositive cross correlations, optimal multiuser detection can be 
done in polynomial time. In this section, we derive a slightly weaker condition than the nonnegativity 
of j, as the necessary and sufficient condition for the success of the LP relaxation to result in an 
integer solution for any value of g in (|9]). This analysis also sheds some light on the question of how the 
algorithm behaves in the general case, where this condition is not satisfied. 

For a check node in the Tanner graph connecting information bit nodes xt and xt-j and state bit node 
zt^j, the constraints (fTTT) can be summarized as 

zt,j > msix[xt - Xt-j, xt-j - Xt] 

zt,j < mm[xt + xt-j, 2 - Xt - xt-j], (12) 
which can be further simplified as 

\xt - xt-j\ < ztj <l-\xt + Xt-j - 1|. (13) 

Since there is exactly one such pair of upper and lower bounds for each state bit, in the solution vector, 
ztj will be equal to either the lower or upper bound, depending on the sign of its coefficient in the linear 
objective function, Xtj. Hence, having the coefficients, the cost of ztj in the objective function can be 
written as 

Xt,j\xt - xt-j\ if Xt,j > 0, 

Xt,jZt,j = < (14) 

[Xt,j - Xt,j\xt + Xt-j - 1| if Xt,j < 0, 
where the first term in the second line is constant and does not affect the solution. Consequently, by 
substituting (fT4]) in the objective function, the LP problem will be projected into the original n-dimensional 
space, giving the equivalent minimization problem 

Minimize f{x) = ^ qtXt + X] '^^'J' ' '^^ ~ ' 

+ \^t,j\\xt + xt-j - ii, 

t,j:Xt,j<0 

Subject to < < 1, Vt = 1, . . . , n, (15) 
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which has a convex and piecewise-linear objective function. Each absolute value term in this expression 
corresponds to a check node in the PR layer of the Tanner graph representation of the channel. 

A. LP-Proper Channels: Guaranteed ML Performance 

For a class of channels, which we call LP-proper channels, the proposed LP relaxation of uncoded 
ML detection always gives the ML solution. The following theorem provides a criterion for recognizing 
LP-proper channels. 

Theorem 1: The LP relaxation of the integer optimization problem (|9]), in the absence of coding, is 
exact for every transmitted sequence and every noise configuration if and only if the following condition 
is satisfied for {Xt,j}' 

Weak Nonnegativity Condition (WNC): Every check node q j, connected to variable nodes xt 
and xt-j, which lies on a cycle in the PR Tanner graph corresponds to a nonnegative coefficient; 
i.e. Xtj > 0. 

Proof: We first prove that WNC is sufficient for guaranteed convergence of LP to the ML sequence, 
and then show that if this condition is not satisfied, there are cases where the LP algorithm fails. In the 
proof, we make use of the following definition. 

Definition 1: Consider a piecewise-linear function / : M"^ i-^ R. We call a a breakpoint of / if the 
derivative of f{a + sv) with respect to s changes at s = 0, for any nonzero vector v G W^. 

1 ) Sufficiency: It is sufficient to show that under WNC, the solution of (fT5]) is always at one of the 
vertices of the unit cube, [0, 1]"^. First consider (fTSl) without the box constraints. The optimum point, x*, 
of the objective function has to occur either at infinity or at a breakpoint of this piecewise-linear function. 
Since the nonlinearity in the function comes from the terms involving the absolute value function, each 
breakpoint, a, is determined by making n of these absolute value terms active, i.e. setting their arguments 
equal to zero. These n terms should have the property that the linear system of equations obtained by 
setting their arguments to zero has a unique solution. 

When the feasible region is restricted to the unit cube [0, 1]"^, the optimum can also occur at the 
boundaries of this region. Without loss of generality, we can assume that the optimum point lies on a 
number, k, of hyperplanes corresponding to the box constraints in (fTSl ). where /c = 0, . . . , n. This will 
make exactly k variables, x^, i e I, equal to either or 1, where |/| = k. In addition, at least n — k 
other equations are needed to determine the remaining n — k fractional variables. These equations will 
be the results of making a number of absolute value terms active, each of them having one of the two 
forms Xt — xt-j or xt + xt+j — 1, depending on whether Xtj > or Xtj < 0, respectively. When an 
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absolute value term in (fTSl ) is active, either both, or none of its variables can be integer. Since the former 
case does not provide an equation in terms of the fractional variables, we can assume that all these active 
absolute value terms only involve fractional variables. 

Now the question becomes under what condition such equations can have a unique and nonintegral 
solution. We can illustrate this system of equations by a dependence graph, where the vertices correspond 
to the unknowns, i.e., the n — k fractional variable nodes, and between vertices Xs and Xs-i there is a 
positive edge if Xg^i > and a negative edge if Ag ^ < 0. An example of a dependence graph satisfying 
WNC is shown in Fig. [3l In the solution of the system of equations, if two vertices are connected in 
the dependence graph by a positive edge, they will have the same value. Hence, we can merge these 
two vertices into a single vertex, and the value that this vertex takes will be shared by the two original 
vertices. If we do this for every positive edge, we will be left with a reduced dependence graph that 
has only negative edges. We claim that, if WNC is satisfied, the reduced graph will be tree. To see this, 
consider a negative edge etj connecting vertex xt on its "left side" to vertex xt-j on its "right side" 
in the original dependence graph. By assumption, etj is not on a cycle, which means if we remove it, 
its left side and right side will become disconnected. Clearly, during the above-mentioned merging of 
vertices, no new connection will be created between the left and right sides of etj . Hence, every negative 
edge will still not be on any cycle at the end of the merging procedure, and, in other words, the reduced 
dependence graph will be a tree. Since trees have fewer edges than vertices, the system of equations for 
determining the unknown variables will be under-determined, and none of the nodes in the dependence 
graph will have a unique solution. Consequently, the only case where we have a unique solution for all 
the variables will be /c = n, which means that all of the variables {xi} are integral. This proves the 
sufficiency of WNC. 

2) Necessity: We prove the necessity of the condition by a counter example. Consider a case where 
the realization of the noise sequence is such that the received sequence is zero. This will make {qt}, the 
coefficients of the linear term in (fTSl) . equal to zero. Hence we are left with the positive- weighted sum 
of a number of absolute value terms, each of them being greater than or equal to zero. The objective 
function will become zero if and only if all these terms are zero, which is satisfied if x = . . . , ^]^. 
We need to show that if WNC is not satisfied, equating all the absolute value terms to zero will determine 
a unique and fractional value for at least one of the elements of x. Consider the dependence graph of 
this system of equations. We know that there is at least one cycle containing a negative edge. Consider 
the equations corresponding to one such cycle. Without loss of generality, we can assume that all these 
equations have the form xt + xt+j = 1, since if any of them has the form xt — xt+j, we can combine 
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Fig. 3. The dependence graph of the system of linear equations with one cluster of cycles. Solid lines represent positive edges 
and dashed lines represent negative edges. 

these two variables into one. Consequently, the system of equations corresponding to this cycle, after 
some column permutations will have the matrix form 

1 1 •• 

110 

1 ••• 1 

Since the coefficient matrix is full rank, the unique solution of this system is = ^, /c = 1, . . . , L This 
means that no integral vector x can make the objective function in (fTSl) zero, and therefore the algorithm 
fails to find an integral solution. This proves the necessity of WNC for guaranteed success of the LP 
relaxation. ■ 
Corollary 1: The solutions of the LP relaxation of uncoded ML equalization are in the space {0, ^, 1}"^. 
Proof: The values of fractional elements of x are the unique solutions of a system of linear equations 
of the forms xt — xt+i and xt + xt+i — 1. The vector [i, . . . , ^] satisfies all these equations, and, hence, 
has to be the unique solution. ■ 

B. Implications of the WNC 

In the PR channel equalization problem, due to the periodic structure of the Tanner graph and the 
coefficients of the state variables, the WNC implies that at least one of the following statements should 
be valid: 

1) The PR Tanner graph is acyclic. Examples include any PR channel with memory size fi = 1, and 
the memory-2 channel h{D) — 1 + D — D^. 









1 




Xi^ 




1 








1 



(16) 
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2) Nonnegativity Condition (NC): All state variables have nonnegative coefficients; i.e. Xtj > Vt, j. 
Lemma 1: Condition 1 implies that, among {A^ j : j = 1, 2, . . .}, on average at most one can be 
nonzero for each t. 

Proof: Let tz be the average number of nonzero elements of {Xtj : j = 1, 2, . . .} for each t. Then, 
it is easy to see that the PR Tanner graph will have S/tn edges and n(l + 2k) vertices (i.e. variable or 
check nodes). But the graph can be acyclic only if S/tn < n(l + 2k,) — 1, which means that k < 1. ■ 

In a one-dimensional ISI channel, where the state coefficients are given by (fTOl) . NC implies that the 
autocorrelation function of the discrete-time impulse response of the channel should be nonpositive at any 
point other that the zero time shift. As the memory size of the channel increases, this condition becomes 
more restrictive, so that a long and randomly-chosen impulse response is not very likely to satisfy NC. 
However, it is very common, particularly in magnetic recording applications, to use the PRML technique, 
where the overall impulse response of the channel is first equalized to a target impulse response, in order 
to simplify the subsequent detection stages. The target channel response is usually optimized to provide 
the best detection performance. A possible approach for employing the linear relaxation of ML detection 
in this application can be to optimize the target channel subject to NC, which enables us to achieve the 
performance of Viterbi detection, although without an exponential complexity in the memory size. 

An interesting application of this result is in 2-D channels, for which there is no feasible extension 
of the Viterbi algorithm. In a 2-D channel, the received signal rt,s at coordinate (t, s) in terms of the 
transmitted symbol sequence {xt^s} has the form 

rt,s = X] X] hi^jXt-i,s-j + nt,s- (17) 

i=0 j=0 

Hence, following the same procedure that results in (fTOl ). the coefficients of the state variables in the 2-D 
channel can also be obtained. In particular, the state variable defined as = xt^s © xt-k,s-i will 

have the coefficient 

lk,i = - ^ ^ hijhi+ij+i. (18) 

i=0 j=0 

In this expression, for simplicity, we have dropped the (t, s) index due to the independence of the 7 
from t and s, except near the boundaries, which, in turn, can be resolved by proper truncation of the 
transmitted array. Theorem [T] guarantees that ML detection can be achieved by linear relaxation if NC 
is satisfied, i.e. jk,i > for any k,l > 0. An example of a 2-D channel satisfying NC is given by the 
matrix 



1 1 

1 -1 



(19) 
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C. Error Event Characterization: asymptotically LP-Proper and LP-Improper Channels 

We showed that if WNC is not satisfied, there are noise configurations that results in the failure of 
LP detection to find the ML sequence. However, for some channels, such noise configurations might 
be highly unlikely at high SNR values. We have observed that for some ISI channels violating WNC, 
the probability of obtaining a fractional solution becomes dominated by the probability that the ML 
sequence is different from the transmitted word. We call these channels asymptotically LP-proper, since 
for these channels the WER of LP detection is asymptotically equal to that of ML detection as the SNR 
increases. On the other hand, for some other channels, which we call LP-improper channels, there is a 
non-diminishing probability that the solution of LP detection is nonintegral, as the SNR increases. In this 
subsection, we study the performance of the uncoded LP detection for general ISI channels and derive 
conditions for failure of the detector to find the ML solution. These conditions will help us to classify 
different channels based on their performance. With some modifications, these conditions can also be 
applied to 2-D ISI channels. Since here we focus on stationary 1-D PR channels, as explained in Section 
II, we can assume that Xtj = Xj is independent of t. 

Definition 2: Given the solution of LP detection, fractional set, F = {zi, . . . , in-k] C {1, . . . , n}, 
is the set of indices of information bit nodes in the Tanner graph of the PR channel that have fractional 
values in the solution, x. 

We know from Corollary [T] that the fractional elements in the solution, x_, are all equal to ^. A 
reasonable assumption supported by our simulations at high SNR is that if the ML solution, x, is correct, 
the integer elements of x are correct, as well. In other words, we have 



By expanding / using (fTSl) . this inequality can be written in terms of {A^ j}, x_, and q. Before doing so, 
we present the following lemma to simplify the absolute value terms in (fTSl) . 

Lemma 2: Let xt and Xs be binary variables and xt and Xg be their bipolar versions, respectively. In 
addition, let's define 




(20) 



For the objective function /(•) of (fT5l ) we can write 



g{x,x) = f{x) - f{x) > 0. 



(21) 




(22) 
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Then, the following equations hold: 

1) h{xt,Xs,X\-t_s\) = ^|A|^_s|| - ^X\t-s\XtXs, 
2' ^\t-s\) = ll-^lt-sll, 



3) ^, X\t-s\) = 0. 

Proof: The equations can be verified by using (dJ, and checking the possible values for xt and Xs- 



By using Lemma [2l and cancelling the terms that are common between f{x) and f{x), g{x,x) can 
be written as 

9{x,x)^Y^ qt(xt-^)+ Yl -\^\t-s\xtxs^ -Y^X\t-s\XtXs , (23) 

teF sEF, s<t s^F 

where = — X^i ^i^i+d is defined to be zero if d > ^. Since we assumed that x is equal to the 
transmitted sequence, we can expand qt as 

Qt = Y Kn+i 

i=0 

= ^ ^ hihjXt+t-j + ^ hiUt+i 

i=0 i=0 i=0 

^ ~ \t-s\Xs + m, (24) 

S=t — fl 

where r]t — Yli ^i'^t+i- By substituting (|24]) into (|23] ). and using the fact that xt — -k = —\xt, we obtain 



^(x, x) 



^Ao + + \\t-s\xtx^ - K]tit 



s^F, s<t 



1 



cf + -^XpPFXp + ri^Xp 



(25) 



In this equation, Xp and 77^ are obtained respectively from x and 77 by keeping only the elements with 
indices in F, Pp is a submatrix of P (defined in II-B) consisting of the elements of P with column and 
row indices in F and its diagonal elements made equal to zero, and 



CF = X!Ao+ 2^2^ \X\t-s\\. 

teF t,sEF, s<t 

Equations (|27I) and (|25] ) lead us to the following Theorem. 

Theorem 2: Uncoded LP detection fails to find the transmitted sequence if there is an index set F C 
{1, . . . , n} for which 

^ ^" ^ (27) 



(26) 



Cf + ^XpPpXp + ri^xp > 0. 
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If the transmitted sequence, x, is given, we can estimate the probability that the sufficient failure 
condition given by Theorem [2] is satisfied, and determine the dominant error event causing this failure. 
In order to do that, for any given F, we can calculate a "distance" for the error event corresponding to 
F defined as 

dp = -cp - -XpPpXp, (28) 
and the variance of the noise corresponding to this error event, given by 



4 



= var^T^^Xj^^j 



= var 



s t=s-)i, tEF 



2 

s t=s-fi, teF 

where cr^ is the variance of each noise sample, nt- Hence, the probability that the error event corresponding 
to F occurs will be equal to 

p{F,Xp) = Q{^), (30) 
ap 

where Q{x) is the Gaussian Q function. 

In order to find the dominant error event over all transmitted sequences, for every choice of the index 
set, F, we should find the vector Xp G {—1, 1}'^' that maximizes the probability in (1301) . However, this 
will require an exhaustive search over all Xp. As an alternative, we can upper bound this probability 
by finding the smallest distance d™™ = min^^ dp and the largest variance cr|.™^^ = rnaxj^ dp, and 
computing Q{df;™/a^^^). Fortunately, each of these two optimization problems can be solved by dynamic 
programming (Viterbi algorithm) over a trellis of at most 2^ states. 

Specifically, if the minimum distance is negative, there is a probability independent of the SNR that 
the sequence Xp corresponding to that distance exists in the transmitted block, and given that event, 
the probability of failure, (f30|) . will be greater than ^ for any SNR value. Therefore, there will be a 
non-diminishing probability of failure as SNR goes to infinity. This motivates the following results: 

Corollary 2 (LP-Improper Channels): If for an ISI channel, there is a index set F C {1, . . . ,n} and 
a vector Xp G { — 1, 1}'^' for which dp defined in ([28]) is negative, LP detection on this channel will 
have a non-diminishing WER as SNR grows; i.e., the channel is improper. ■ 

On the other hand, if for an ISI channel the probability of failure computed by the proposed technique 
decreases more steeply than the WER of ML detection as SNR increases, the WER of LP detection 
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will be asymptotically equal to that of ML detection. In this case, the channel will be asymptotically 
LP-proper. 

Remark 1: The error events considered in this analysis are not the only possible types of detector 
failure. This analysis is intended to approximate the gap between the performance of LP and ML detection 
methods by estimating the probability that LP detection fails to find the integer optimum (i.e., ML) 
solution, given the ML detector is successful. As mentioned before, we only studied the events where a 
vector having the form of (|20]) has a lower cost than the transmitted word. Therefore, even if dTTI ) does 
not hold for any F, it is theoretically possible, although not very likely, as we observed in practice, that 
LP detection has a fractional solution. 

Of particular interest among the possible error events is the one where the all-| vector has a lower 
cost than the correct solution; i.e., F = {1, . . . ,n} in (ITtI) . For a given transmitted sequence x, this 
event is not necessarily the most likely one. However, studying this event provides us with a simplified 
sufficient condition for the failure of LP detection, which further clarifies the distinction between the 
different classes of ISI channels. 

The distance, 5, corresponding to this event is obtained by putting F = {1, . . . , n} in (|28]) . If the block 
length, n is much larger than the channel memory length, (i, we can neglect the "edge effects" caused 
by the indices that are within a distance n of one of the two ends of the sequence; thus, we will have 

8 = -nAo - l^jl - Aj xtxt+j 
j=i j=i t 

= -nXo - n^\Xj\ -^XjPj, (31) 

3=1 3=1 

where pj is the autocorrelation of x with a shift equal to j. On the other hand, for the noise variance, 
(^^, corresponding to this event we have from (|29]) 

r ^ 

<^^^a'^J2 ^hjXt-j 

t L j=Q 

= a^x^Px 

= -a^nXo - 2(7^ ^ Xjpj . (32) 

3=1 

Note that 5 and have a similar dependence on the transmitted sequence. 

A possible approach for finding the likelihood of occurrence of an all-^ error event is to maximize | 
over all possible transmitted sequences. However, with a random transmitted sequence, the probability 
that this quantity becomes close to its worst case may become very low for long block lengths. As an 
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alternative, here we show that as n grows while n remains fixed, the dependence of both 5 and on 
the transmitted sequence become negligible compared to the constant term. 

Lemma 3: Let be a sequence of i.i.d. ±1 random variables, each equally likely to be +1 

or —1, and let pj = Yl't~'' ^tXt+j- Then, for fixed ^, as n ^ oo 



1 

— XjOj almost surely. (33) 
n ^-^ 

3=1 

Proof: For each j = 1, p, pj is the sum of n — j terms of the form xtXt+j. Clearly, each of 
these terms is equally likely to be equal to +1 or —1. Furthermore, it can be shown that these terms are 
mutually independenj^. Hence, using the strong law of large numbers, we have 

n-j 

n n \n — 1 

^ t=i 

where we used the fact that n — j/n ^ 1, since 1 < j < p. Consequently, 



1 n--j 

— = ( xtXt+j ) almost surely, (34) 



— Xjpj = A, — ^ almost surely, (35) 
n ^-^ ^-^ n 

since it is a linear combination of a finite number of variables, each going to zero almost surely. ■ 
Using Lemma [3] in (|3T] ) and ([32]) for large n, we can write 

^ — n 



(36) 



and 



<^2 = a2n[|Ao|+o(l)], (37) 

where we used the fact that Aq = — X] ^ 0- Since the probability of the all-^ error event is equal to 
Q{5/q), the above results motivate us to define the following parameter to characterize this probability 
in the limit of n ^ cx). 

Definition 3: The LP distance, (5oo, of a partial-response channel is given by 

^oo= p-(|Ao|-^|A,|). (38) 

j=i 

The LP distance is dimention-less parameter that can take values between — oo and 1. The following 
theorem gives a new sufficient condition in terms of the LP-distance for a channel to be LP-improper. 

Theorem 3: The WER of uncoded LP detection over an ISI channel with the transmitted sequence 
generated as a random sequence of i.i.d. Bernouli(l/2) binary symbols goes to 1 as the block length n 

'For a proof of this statement refer to Proposition 1.1 of [11]. 
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goes to infinity for any SNR, i.e., the channel is LP-improper, if the LP distance, ^oo^ of the channel is 
nonpositive 

Proof: As mentioned earlier, the probability of the all-| event is equal to Q{6/<;). From (|36]) and 
([37]) . for large n we have 

- = ^/^(^oo^^ + o(l)). (39) 

If Soo < 0, the right-hand side will approach — co as n increases, hence, Q{5/q) will go to 1. ■ 
Having ^oo as a measure of LP-propemess for LP detection, it is interesting to study how it behaves for 
LP-proper channels; i.e., those satisfying WNC in Theorem [H The following lemma provides an answer 
to this question. 

Lemma 4: For LP-proper channels that satisfy NC (defined in in III-B) > ^\Xq\. 
Proof: For any ISI channel, we can write 

= |Ao|-2^A, >0. (40) 

Hence, 

EA,<i|Ao|. (41) 
Since NC is satisfied, Xj > 0, Vj > 1. Therefore, we have 

<^oo = |Ao| — Aj 



J=l 

> ^|Ao|. (42) 



D. Simulation Results 

We have simulated channel detection on the PR Tanner graph using LP decoding and MSA for three 
PR channels of memory size 3: 

1) CHI: h{D) = 1-D- 0.5L>2 _ o.5L>3 (with ^^o = ^ satisfies WNC; LP-proper), 

2) CH2: h{D) = 1 + D - + (with 6oo = 5), 

3) CH3: h{D) = l + D-D'^-D^ (with 5oo = 0; LP-improper). 
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Fig. 4. BER for CH1-CH3. SNR is defined as the transmitted signal power to the received noise variance. 

Uncoded bit error rates (BER) of detection on these channels using LP and MSA are shown Fig. |4] Since 
CHI satisfies WNC, LP will be equivalent to ML on this channel. For CH2, we have also provided the 
BER of ML. Except at very low SNR where we see a small difference, the performance of LP and ML 
are nearly equal, which means that CH2 is an asymptotically LP-proper channel. For both CHI and CH2, 
MSA, converges in at most 3 iterations and has a BER very close to that of LP. On the other hand, for 
CHS, which is an LP-improper channel, we observe that the BERs of LP and MSA are almost constant. 

In Fig. m we studied the effect of ^oo of ISI channels on the performance of LP detection. In this 
scenario, we randomly generate 200 ISI channels with memory 4, such that the taps of the impulse 
response are i.i.d. with a zero-mean Gaussian distribution. In addition, we normalize each channel so that 
the total energy of the impulse response is one; i.e. |Ao| = 1. We simulated the uncoded LP detection 
with random transmitted sequences of length 100 at the SNR of 11 dB. In the upper and middle plots 
of Fig. [5j respectively, the WER of LP detection and the ratio of the WER of LP detection to that of 
ML detection are shown versus 6oo for these channels. In this work, ML detection was performed by 
using the cutting-plane method proposed in [12]. This method is based on using redundant parity checks 
(RPC) generated by modulo-2 combination of a subset of parity-check constraints. Once the LP decoding 
results in a nonintegral solution, we look for RPCs that, after linear relaxation, introduce a cut, i.e., make 
the current solution infeasible, and re-solve the LP after adding these constraints. This is continued until 
we obtain an integral solution, which is the ML sequence. For 192 of the 200 channels that we studied. 
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Fig. 5. Upper and middle plots: Performance of LP detection versus Sqo for random ISI channels of memory 4 at the SNR of 
11 dB. Lower plot: The histogram of 5oo- 

this algorithm always successfully provided the ML solution with a few iterations. 

The results in Fig. [5] demonstrate a strong correlation between the performance of the algorithm and the 
value of ^oo- In particular, almost every channel with ^oo < 0.1 has a WER close to 1, while for almost 
every channel with > 0.4, the WER of LP is very close to that of ML detection. We have observed 
that detection by MSA had a similar behavior, except for some channels with 0.05 < ^oo < 0.3, for 
which MSA is significantly superior to LR In other words, the transition from LP-improper to LP-proper 
behavior starts from smaller values of ^oo for MSA. 

As an estimate of the probability density function of ^oo for this random construction of the channel 
response, its histogram has been included in the lower plot of Fig. [H 
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IV. Combined Equalization and LDPC Decoding 

One of the main advantages of the graph-based detection proposed in Section II is that it lends itself 
to the combining of the equalization with the decoding of an underlying error-correcting code. In this 
section, we study this joint detection scheme using both the linear programming and the MP algorithms. 

A. Coded Linear Programming Detection 

Joint LP equalization and decoding, i.e., the linear relaxation of ([9]), is a linear optimization problem 
in the form of the uncoded LP detection, with the addition of the linear inequalities corresponding to the 
relaxation of the parity-check constraints of the code. These new constraints cut off from the feasible 
polytope some of the fractional vertices that can trap the uncoded detection problem, but they also add 
new fractional vertices to the polytope. It is not easy to derive general conditions for the success or 
failure of this problem. However, we can make the following generalization of Theorem [3] 

Corollary 3: Consider a linear code with no "trivial" (i.e., degree- 1) parity check, used on a channel 
satisfying Soo < 0, where ^oo is defined in (??). Then, coded LP detection on this system has a non- 
diminishing WER for large block lengths. 

Proof: We have shown in Section III-C that if this condition is satisfied, the all-^ vector will have 
a lower cost than the transmitted vector with high probability. It is now enough to show that the all-^ 
vector will not be cut off by any error correcting code. To see this, consider a relaxed parity-check 
inequality of the form 

J2^^- V y C TVc s.t. is odd, (43) 

where Nc > 2. To prove that this constraint is satisfied by the all-^ vector, we consider two cases: 
\V\ — 1, and |y| > 2. If |y| = 1, the first sum in (|43] ) will be equal to 1/2, and the second sum will be 
greater than or equal to 1/2, since Nc\V has at least one member. Hence, the left-hand side of (|43]) will 
be less than or equal to |y| — 1 = 0. Also, if |y| > 2, the first sum will be equal to \V\/2 < \V\ — 1, 
while the second sum is non-negative. Therefore, the left-hand side of the inequality will be less than or 
equal to its right-hand side. Consequently, in both cases (|43]) will be satisfied. ■ 

^The derivation of the limit of 6 was based on the assumption that the transmitted sequence is an i.i.d. sequence, so that ((3) 
holds. While the transmitted sequence is no longer i.i.d. in the presence of coding, we implicitly assume that OJl still holds. This 
is a sufficiently accurate assumption for all codes of practical interest. In particular, ((3) can be proved for a random ensemble 
of LDPC codes. 
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B. Coded Message-Passing Detection 

Similar to LP detection, MP detection can be extended to coded systems by adding the parity-check 
constraints of the code to the PR Tanner graph, as shown in Fig. [2l and treating it as a single Tanner graph 
defining a linear code. Despite many similarities, LP and MP decoding schemes have a different nature, 
which makes them behave differently when used for joint equalization and detection. For example, we 
cannot derive a conclusion similar to Corollary [3] for MP detection. On the contrary, we have observed 
in the simulation results that there are LP-improper channels for which coded MP detection does not 
inherit the undesirable performance of uncoded MP detection. 

In this work, we use both the min-sum algorithm and the sum-product algorithm (SPA) for the 
implementation of coded MP detection. Similar to the uncoded case, as the objective coefficients of 
MSA, we use the same coefficients as those of LP detection, i.e., {qt\ and since MSA is invariant 

under the scaling of the coefficients. For SPA, we observe that each qt contains a Gaussian noise term 
with variance proportional to cr^. Hence, one can argue that a suitable normalization of the objective 
coefficients to estimate the "equivalent LLRs" is to multiply all the objective coefficients by 2/cr^. An 
advantage of this normalization is that, in the absence of ISI, the normalized coefficients become the 
true LLR of the received samples. In this work, we have used the equivalent LLRs obtained by this 
normalization for SPA detection. 

MSA and SPA decoding are, respectively, approximations of ML and a posteriori probability (APP) 
detection on the Tanner graph defining the code. These approximations becomes exact if the messages 
incoming to any node are statistically independent. This happens if the Tanner graph is cycle-free and 
the channel observations (i.e., the a priori LLRs) are independent. In the proposed graph-based detection 
neither of these two conditions is satisfied. In particular, the PR layer of the graph contains many 6-cycles, 
and the channel observations, {qt], are the results of matched filtering of the received signal, and thus 
contain colored noise. 

In order to mitigate the positive feedback due to the cycles of the PR layer, we propose selective message 
passing for coded detection, which has a modified combining rule for messages at the information bit 
nodes. This modified combining scheme is illustrated in Fig. [6l In this method, the message outgoing 
from an information bit node through an edge e in the code layer is computed as a combination of the 
channel observation and the messages incoming to the bit node through all edges, except edge e. On 
the other hand, the message outgoing through an edge in the PR layer is a combination of the channel 
observation and messages incoming through only the edges in the code layer. Since there are no 4-cycles 
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Fig. 6. Selective message passing at the information bit nodes: (a) Calculating a message outgoing to the code layer (b) 
Calculating a message outgoing to the PR layer. 

in the PR layer of the graph, this modification blocks any closed-loop circulation of messages inside the 
PR layer. In other words, message passing inside the PR layer will become loop-free. However, there 
still remain cycles in the code layer, as well as cycles that are generated by combining the code and PR 
layers. In our simulations, selective MP performed as an effective tool for improving coded MP detection 
of some channels with undesirable properties, such as the EPR4 channel. 

C. Simulation Results 

In this subsection, we present simulation results of coded detection in the presence of ISI using the 
schemes proposed in this section. In all cases, we have used a regular LDPC code of length 200, rate 
1/4, and variable degree 3. The following PR channels have been used in these simulations: 

1) No-ISI Channel: h{D) = 1, 

2) EPR4 Channel: h{D) = 1 + D - D"^ - (Soo = 0, LP-improper), 

3) Modified EPR4: h{D) = I + D - D'^ + {5oo = \, asymptotically LP-proper), 

4) PR4 Channel: h{D) = 1 - {5oo ^ \, LP-proper). 

In Fig. |7J the BER of coded LP and MSA detection has been plotted versus -E^/A^o for the above 
channels. For all channels, except in the ISI-free case, coded MSA detection outperforms coded LP 
detection. In particular, for the EPR4, coded LP detection has a BER of about 1/2 for all SNR values, 
as predicted by Corollary [3l while coded MSA detection has a monotonically-decreasing BER. 

To study the behavior of the different detection methods for the EPR4 channel in more detail, in Fig. 
H the BER has been plotted versus E^^jNo for LP, MSA, selective MSA, SPA, and selective SPA. One 
can observe in this figure that selective message passing is mostly effective for MSA, for which there is 
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Fig. 7. BER vs. E^/No for coded LP detection and coded MSA detection in four channels. 



a 0.5 dB SNR gain. In addition, by using SPA instead of MSA, we obtain a 2 dB SNR gain. We have 
observed that, for the other three channel, the gap between MSA and SPA was between 0.3 to 0.7 dB. 

V. Conclusion 

In this paper, we introduced a new graph representation of ML detection in ISI channels, which can be 
used for combined equalization and decoding using LP relaxation or iterative message-passing methods. 
By a geometric study of the problem, we derived a necessary and sufficient condition for the equalization 
problem to give the ML solution for all transmitted sequences and all noise configurations under LP 
relaxation. Moreover, for certain other channels violating this condition, the performance of LP is very 
close to that of ML at high SNRs. For a third class of channels, LP detection has a probability of failure 
bounded away from zero at all SNR, even in the absence of noise. In a step toward the analysis of the 
performance in the general case, we derived a distance, ^oo, for ISI channels, which can be used as a 
tool to estimate the asymptotic behavior of the proposed detection method. Simulation results show that 
message-passing techniques have a similar performance to that of LP for most channels. In addition, 
we studied graph-based joint detection and decoding of channels with LDPC-coded inputs. Simulation 
results indicate that, in contrast to the uncoded case, message-passing detection significantly outperforms 
LP detection for some channels. 
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Fig. 8. BER vs. Eb/No for various coded detection schemes in the EPR4 channel. 
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